The Archives portal Europe (www.archivesportaleurope.net) publishes the archival contents provided by its different partners through a back-office called the Dashboard. An option allows delivering the data also to Europeana by using the same Dashboard.
This document describes the different tasks to be done by the institution manager within the Dashboard.
Three levels of administrators exist in the Dashboard: general administrator, country manager (CM), institution manager (IM).
The general administrator creates the country manager accounts within the Dashboard when asked to do so by a country and manages the CM accounts. He can take the lead on any account in case of problem.
The country manager (only one per country), generally appointed by the National archives but not always, has specific access and rights in the Dashboard, allowing him to:
The institution manager (only one per institution, but one institution manager can be in charge of different institutions) has specific access and rights in the Dashboard, allowing him to:
Figure 1: Main menu of the institution manager's Dashboard:
- Content provider information to manage his institution and the contacts received
- Content information to upload and manage his finding aids and EAC-CPF files
The Archives Portal Europe offers two environments to the partners: the Production and the Content Checker. Each environment consists of a front-end (the portal) and a back-office (the dashboard). They are strictly independent from each other.
- The Production Dashboard
The Production Dashboard (http://www.archivesportaleurope.net/Dashboard) is the web environment where you upload your data for the online public production version, available for the public and the search engines.
- The Content Checker Dashboard
The Content Checker Dashboard (https://contentchecker.archivesportaleurope.net/Dashboard) is a simulation environment of the former, almost exact duplicate of the production one, but not visible by the public and search engines.
Figure 2: Two environments: Content Checker and Production
This simulation environment allows you to experiment the portal, test the workflow of data upload and processing, check how your data will look like on the portal before publishing them “for real” in the production environment. Needless to say that it is highly recommended to use the Content Checker before using the Production Dashboard.
The Dashboard includes all necessary functionalities for the Institution managers to:
All source files submitted to the Archives Portal Europe should be in XML format and at least compliant with one of the following standards:
It is highly recommended to use UTF-8 encoding.
The portal relies on different types of XML files: EAG files to describe the institutions, EAD files to describe the archives, EAC-CPF to describe the persons. They are all based on the archival international standards. Europeana uses another type of XML file, the EDM files. To these different types of files correspond different processes and workflows.
The Dashboard is accessible after email registration. Once appointed by your institution for the role of the institution manager, you receive an email containing your username (which is your email address), a password, a secret question and answer and a link to directly access the Dashboard. If your institution has already signed the Content Provider Agreement (available via the Country Manager, a list of Country Managers can be found over here), you will get an access to the Content Checker and to the Production environments through two different messages. Otherwise, you can access to the Content Checker environment only.
Figure 3: Login screen of the Dashboard
note that the environment is always mentioned in the header (here: Production)
Once logged in, you can change in the “edit user information” menu your name, surname, secret question and answer and password (strongly recommended), but not your email address.
If you forget your password, you can get a new one by clicking on the link mentioned in the login page and from then follow the instruction that you will receive via email.
You can also change the language of the interface by using the dropdown list on the top right corner of the Dashboard.
Figure 4: Edit user information
Your institution has a specific place in the archival landscape of your country, given by your country manager who manages it. This archival landscape is the entry point to the 3-layer structure -country, institution, archival content- of the portal. It is the first level of this hierarchical structure, on which is based the navigation tree structure of the advanced search as well as the directory of the portal.
The menu "Content provider information" allows you to manage your institution description which is done in an EAG file. All information about the EAG can be found over here. The schema is available at: http://www.archivesportaleurope.net/Portal/profiles/eag_2012.xsd.
Figure 5: Menu Content provider information (when the EAG is already created or uploaded)
The EAG file can be either uploaded to the Dashboard, either created within the Dashboard via a specific form. When first logged-in, you have to upload or create your EAG file in order to appear in the directory and to be allowed to upload finding aids and other contents.
If you already have an EAG file (either from your own environment, or created within the Data Preparation Tool), you can upload it in the Dashboard. The Dashboard will check if the file is compliant, and in case some mandatory information is missing, you will be asked to fill them in before saving the file.
To create the EAG file, you have to fill in a form, consisting of 7 tabs, ordered as in the ISDIAH standard, on which the EAG is based: Identity, Contact, Access and services, Description, Control, Relations. The mandatory fields are all grouped on the first tab, and the name of your institution is pre-filled.
Figure 6: EAG creation form
To have a complete description and all explanation, please refer to the EAG guide that describes each element: http://www.apex-project.eu/images/docs/D2.6D4.8_mapping-guide_apeEAD.pdf. A specific manual on how to complete the form is also available in the project website: http://www.apex-project.eu.
Specific attention should be given to the identifier of your institution. If you do have an ISIL code please use it. If you don’t, an internal identifier is automatically generated by the Dashboard. If you do use unique identifiers within your country, and that are ISO 3166 compliant, you can also use it. The ID should consist of two characters for the country code, a hyphen and up to 11 alphanumeric characters. For instance:
The mandatory information filled in the first tab are automatically reported in the corresponding tabs. For instance, the name in the Identity tab or the opening hours in the Access and service tab. If your institution runs several repositories, for instance the headquarters and different branches, you can add them one after the other by clicking on the "Add new repository" button. You have then to fill-in the same information as for the main repository. Each repository is then managed in its specific tab, and the display in the directory shows all of them.
Figure 7a: Dashboard view of an institution having two repositories
Figure 7b: Portal view of the same institution (information folded)
Though possible and easy to achieve, the change of name of your institution should be handle very carefully. Besides, you should always report such changes to your Country manager. Please note that the change of name can be done only if no content is published in the portal or converted to EDM. Therefore, you have to unpublish and republish erverything after the change, including the EDM files.
Figure 8: Form to change the name of your institution
The users have the possibility to contact the institutions having published their content on the portal through a form, available at each level. By default, the institution manager receives these emails, but another email can be preferably chosen.
Figure 9: Change the email address.
Please note that this field is mandatory: if left blank, the institution manager's email address will be used automatically.
In order to better run the portal, namely for searching and displaying issues, a common EAD profile has been defined, named apeEAD, as well as a common EAC-CPF profile, named apeEAC-CPF. All information relative to these profiles can be found over here and the profiles themselves can be found over here: http://www.archivesportaleurope.net/Portal/profiles/apeEAD.xsd and here: http://www.archivesportaleurope.net/Portal/profiles/apeEAC-CPF.xsd. Unless already compliant to them, the original local files have thus to be converted to these specific schemas before being published. The portal hosts different types of EAD files: finding aids, holding guides and source guides. There is a hierarchical relation between the holdings guide and the finding aids that is materialised in the search tree of the advanced search page of the portal.
Figure 10: Hierarchy between the holdings guide and the finding aids
From the search tree, you can access either one specific finding aid, or the holdings guide
There are also relations between the EAC-CPF files and the FA published in the portal to allow the users to move from one to the other easily thanks to internal links displayed in the "Archival materials" facet (for more detail on EAC-CPF, please see chapter 5.1).
Figure 11: Links from the EAC-CPF file to the finding aid
As said in chapter 1.3, only XML files can be uploaded in the Archives Portal Europe. These can be database exports or a copy of existing EAD/XML files. During the export, a mapping could be needed to a local XML format, to a target schema such as EAD 2002 or directly to the EAD profile defined for the Archives Portal Europe (apeEAD). It is wise to collect all files in one place (e.g. one folder) which would be of help for submitting the data in one-go, f.i. when intending to use either an OAI-PMH repository or a FTP server to upload files or when wanting to upload several files combined in a ZIP folder via HTTP.
It is highly beneficial for the archives to think ahead of a bigger picture: the data ecosystem on the web. It is very likely that the content providers have to take care of the integrity of their own data on the internet as they increasingly tend to publish the data through various channels including their own website(s), third party websites (e.g. international, national, and regional portal, thematic portal etc.), and Linked Open Data publication.
The issue of data integrity is that the content providers continuously update their source data, implying the data available through different channels also needs to be updated and/or give feedback to the original data source.
Some tips are given below to better take this into consideration:
In order to facilitate the work of the institution managers, the Dashboard provides the possibility to establish specific profiles. These profiles, mandatory when using the OAI-PMH functionality, allow to automatically process the data once uploaded in the portal. It is particularly useful in the case of regular update and addition of content in the portal. However, it is recommended to first test manually the portal in order to better see the different possibilities and check what is best for your data.
The profiles are used in the Dashboard to indicate which actions are to be applied to the uploaded files. You can create as many profiles as needed, for instance you could apply different rules for files without images, and files containing links to images. When using the OAI-PMH harvesting, using a profile is mandatory. Please note that you can create a "manual" profile that will allow you to process the data yourself, step after step.
When you create a profile, you have to give the profile a name, and precise the type of file (finding aid, holdings guide, source guide or EAC-CPF) it is associated to and the forms will be adapted accordingly. Then you can indicate your choices in two different tabs: preferences for the Archives Portal Europe (tab "Basic preferences" displayed by default) and preferences for Europeana.
Figure 12: Create a profile: basic preferences tab, here for EAC-CPF files
The basic preferences indicate the default actions to apply to your files:
Specific options can be provided for the FA, regarding the rights and the dao type. Note that this can be also added afterwards manually, file per file, from the content manager, when clicking on "options" in the column "converted".
Figure 13: Create a profile: precise the rights statements for an EAD file
For Europeana, the EAD files have to be converted to another format (EDM), totally different from the EAD format, and are published in another portal that has different re-use rules than the Archives Portal Europe. The preferences to indicate are therefore numerous and subdivided into general and specific settings.
As converting an EAD file to the EDM standard means "flattening" the description of the document, the main options are related to the information that you want to report from the high levels of description into each EDM record. Note also that you can choose between two types of conversion: the minimal and the full one. The minimal will only take some basic elements such as the unittitle. For more information, please see chapter 5.5.
Please note that you will be allowed to forward content to Europeana only if you have signed the Europeana Data Exchange Agreement (DEA).
Figure 14: Create a profile: Europeana preferences tab
The Dashboard allows three different protocols to upload the files: HTTP, FTP and OAI-PMH. HTPP and FTP are available by choosing "upload content" in the main menu, OAI-PMH by choosing "create automated harvesting function".
Short overview of the pros and cons of each method:
OAI-PMH | FTP | HTTP | |
Pros |
|
|
|
Cons |
|
|
|
Figure 15: HTTP protocol for uploading the files
Before actually upload the file, you can choose a profile to apply specific actions
You may use a predefined profile or not. For the profiles, see chapter 4.2.
You can upload one xml file, or one zip file containing several xml files. Your file has to be a valid xml file or a zip file containing valid xml files. If the files are not valid, the Dashboard will reject them, and produce a report (see chapter 4.4.2). The size limit of the file (xml or zip) is 200 MB. Once in the Dashboard, you have to indicate whether the uploaded files are holdings guides, finding aids, source guides or EAC-CPF files. Depending on the creation process of your files, the Dashboard will detect the type automatically or not. You can change the type via the drop-down list next to each file.
Figure 16: Upload of date: check the type of files
Once these checks and validation are done, you are invited to go to the Content manager.
Please note that if you use a profile, all steps apart the upload itself can be managed by the Dashboard without any further manual contribution.
The number of files during this check step is limited to 500. If you uploaded a zip containing more than 500 files, you will have to repeat this step as many times as necessary.
The process is similar to the HTTP. When choosing the FPT protocol in the upload content menu, you have to fill in the address of your FTP server, give the username and the password, and connect to the FTP server. The profile to apply can be selected afterwards, when you select the files to be uploaded.
Figure 17: Upload of data via FTP protocol: first step, connecting to the server
Use the OAI protocol is highly recommended . For more information, please refer to the OAI-PMH website: http://www.openarchives.org/pmh. You can also read the Best practice for OAI PMH Data Provider Implementations and Shareable Metadata (a bit old, but the bases of the OAI-PMH did not change): http://old.diglib.org/pubs/dlf108.pdf. Once set up, everything can be automated, from the harvest to the publication of data in the Portal and delivery to Europeana.
There are many open source OAI-PMH tools (for more details see: http://www.openarchives.org/pmh/tools/tools.php. When implementing an OAI-PMH repository, it is recommended to test it before submitting data to the Archives Portal Europe. There are several (online) testing tools that can also be used for this purpose (e.g. OAI repository Explorer, see: http://re.cs.uct.ac.za and Data Provider Validation and Registration, see: http://www.openarchives.org/Register/ValidateSite).
Figure 18: Result of a check on the repository itself made on http://www.openarchives.org
Some important point have to be checked to ensure a correct harvesting process and take advantage of all its possibilities:
Figure 19: General schema of the harvesting syntax
In the Dashboard, you act as harvester from your own repository. The advantage is that you fully control the harvest process. The first step is to enter the URL of your OAI repository to allow the Dashboard to check it.
Figure 20: URL of the repository
The Dashboard checks the repository and adapts accordingly the form to allow you to set up the harvest
The second step consists to precise the parameters of the harvest, based on what the answers made by your repository to the requests of the Dashboard:
Figure 21: Harvest parameters
When the harvest is terminated and successful, the files are available in the Content Manager and, depending on the profile chosen, ready to be further processed, or already converted, published, delivered to Europeana and so on. In parallel, you receive an automatic message from the Dashboard informing you of the result of the harvest: if the harvest succeeded or not, how many files could be harvested, and a short text describing the problem, if any. You can then check in the Portal and in the Dashboard how your data have been handled and displayed and get details on the errors (see next chapter).
The harvests are presented in a table summarizing for each of them the parameters chosen, the result of the last harvest and allowing their edition and the download of files (by clicking on the texts in the "errors" column). After a harvest you have four possibilities: succeed, succeed with warnings, succeed with errors and failed, indicated with a color code, green, orange and red.
Figure 22a and 22b: Status of the harvests
Except for the first case, you can download files to check what happens.
Record 'oai:{reference of the record}' (2013-05-22) is deleted in OAI-PMH repository. Please delete it manually in the dashboard
Please note that only the first encountered error is commented. This does not mean that there are no other errors after.
The APEx project developed a standalone console allowing to harvest your data as you would in the Dashboard, using the same parameters etc. This tool can be downloaded over here.
It’s useful in different cases:
When uploaded, the files are "passed, ie read by the machine to check whether they are compliant with the defined standards and rules of the portal (xml or zip, valid, already in the portal or not).
When the files are uploaded and the data types confirmed, the user always sees the list of uploaded files categorised as follows:
Figure 23: Status of the uploaded files: three possibilities
Upload is complete without any problems. Clicking on the “accept” button at the bottom of the screen will achieve the upload process and integrate the files in the content manager.
Only valid xml files can be accepted in the portal. If the files are not valid, an error message will be displayed when clicking on "click for more information" (see example below).
Figure 24: Error message: the error is explicitly indicated, with the number of the row and column
Here, the namespace is incomplete
<eadid> is the unique identifier of an EAD file used in the Archives Portal Europe. Accordingly, if the uploaded files have the same <eadid> as existing files in the Dashboard, they are regarded as duplicates. The same goes for the <recordId> for an EAC-CPF file.
Three possibilities are then offered to the user: overwrite, change the ID manually, cancel the upload.
The existing file is deleted from the index and replaced by the new one. The corresponding EDM files are also deleted and a "delete" tag is assigned to the EDM records stored in the OAI-PMH repository for Europeana, so that Europeana will be able to remove the records at the point of the next harvesting (once a month).
The uploaded file will be rejected and the existing file will be kept in the Dashboard.
The ID, identifier of the file, is either the <eadid> for the EAD files, or the <recordId> for the EAC-CPF files. In order to change the ID of the file, it is necessary to check the availability of the new ID provided. If the users fail to do it, the files will not be recognised, thus not be ingested further. No error message will be displayed for this case.
Figure 25: A new ID (in this case <recordId>) has to be filled in and checked
Please note that the "Accept" button at the bottom of the list is valid for the whole page (and files).
The Content manager allows you to apply all needed actions on your files. The section is divided in three parts, interrelated, framed in different colors in the figure below.
Figure 26: Overview of the Content manager
Figure 27: Selecting files
The different actions or steps are the following:
For an uploaded finding aid, you may convert it to apeEAD, validate it, apply a profile to it or delete it. For a converted finding aid, you may edit it validate it, see the error message, download it, apply a profile or delete it. For a validated finding aid, you may publish it, edit it, preview it, download it, see the error message, see the EAD data quality report, apply profile, convert it to EDM, add it to holdings guide or source guide or delete it. Etc.
For one file: useful to check it when an error occurred, or when you are not satisfied with the final display in the portal. You have to use the download action in the dropdown list of the main table of the Content manager (see figure 26).
For several or all files: useful for testing purposes, or if you want to pass from the content checker to the production environment, and also if you want to participate to another project that has similar requirements. You have to use a specific functionality developed on purpose with Webdav:
If you are not already logged in the Dashboard, you will be asked to do so: please provide the same login and password as for the Dashboard.
The files are hierarchically organised in folders by country, institutions and type of files. You can download the contents from any level. Depending on your status (Country manager or Institution manager) you have different right access.
Figure 28a: Country level: here landing page for the Country Manager of France
Figure 28b: Institution level: when clicking on the country level for a CM,
or when managing several institution for an IM
For each institution, the files are organised by type of files, FA, SG, HG, EAG, EAC-CPF and EDM, ESE-HTML for Europeana. At the lowest level, clicking on the name of a file will display it in your browser. To actually retrieve the files, you have to use software, like cyberduck for instance (a FTP platform).
Figure 29:Browsing the folders to choose the data to download (here from CM level, using cyberduck)
Different options are available in the Dashboard:
An introduction to Encoded Archival Context - Corporate Bodies, Persons and Families (EAC-CPF) can be find on the apex-project website and the schema is available on the portal: http://www.archivesportaleurope.net/Portal/profiles/apeEAC-CPF.xsd.
Content providers, who cannot export EAC-CPF files from their own system, or don't have any yet, can create these files in the Dashboard with a form. The form does not however cover all possible elements of apeEAC-CPF, but allows entering the key information about an entity. The created files are then manageable in the content manager, like the other files (please see chapter 4.5).
Creating an EAC-CPF file is done in two steps: first select the type of entity that you want to describe, then fill in the different tabs adapted to each type.
Figure 30: Create an EAC-CPF file. Step one
Choose the type of entity-person, corporate body or family
Figure 31: Create an EAC-CPF file. Step two
Fill in the form with all available information
The quality of the search in the portal and the facets to refine the search depend on the data available in the EAC-CPF file.
Figure 32: Search results for EAC-CPF
Display of an EAC-CPF file in the portal: external links to the original websites and internal links to the finding aids where the entity can be found are available in the box on the right, named "archival materials":
Figure 33: EAC-CPF file as displayed in the portal
The topics populate the tag-cloud displayed on the homepage or the portal, and are also available in a list via the menu "topics" in the footer. They are used as facets in the search results.
Figure 34a: Tag cloud
Figure 34b: List of topics in the portal
There are two ways to activate the topics in the Archives Portal Europe:
Figure 35: General table to manage the topics
The "topic description" contains the original (English) term that is displayed in portal and that you can translate in the project "topics" of Archives Portal Europe Translation Tool.
The "keyword for controlaccess" is the place to map your own terms with the topics of the portal. In the example above, the topic First World War (1914 -1918) is mapped to the terms "Première guerre mondiale" and "Grande Guerre", terms that are used in the finding aids.
The "source guide" column is automatically filled in with the source guide linked to one (or more) topic.
You can order the column, modify or delete an existing topic or create a new topic mapping.
Figure 36: Create a new topic mapping
You have two dropdown lists at the top:
Figure 37: Example of mapping for "taxation" for the French National Archives:
11 terms have been found so far and could be completed in the future, if new terms were to be used
The terms are case insensitive: Taxation or taxation will be index equally.
The limitation of characters is set to 1000. It allows entering a lot of synonyms in the case of very general topics that do corresponds to numerous specific terms.
Please note that it takes around 10 to 15 minutes before "your" topics are taken into account in the tag cloud
To help you map the topics with your existing index entries or thesaurus, a new functionality has been developed in the Data Preparation Tool. You can list all <subject> element of all your files in one go, and see which are the most relevant and important and could be mapped to the existing list of topics of the portal.
There are two possibilities to edit the finding aids in the Content Manager of the Dashboard: via the Options of the converted file (also available in the preferences, see chapter 4.2 of this document) or via the "Edit" button (see chapter 4.5).
The following elements can be modified in order to improve the data "quality" or the display of the finding aids:
Figure 38: Edit EAD file functionality: the full tree of the file is displayed.
The editable elements are shown on the right once a level is selected in the tree
Please note that in order to edit a file, you need to first unpublish it - if it had already been published.
The Dashboard offers the possibility to create basic holdings and source guides.
The holdings guide offers high level descriptions of all fonds and links to the detailed finding aids, when these are available. It is the second layer of the three-layer structure of the contents: institutions / holdings guides / finding aids. If an institution doesn’t have any finding aids, creating a holdings guide can be a way to nonetheless describe the fonds and the filing plan for instance.
The source guide, when created within the Dashboard, allows selecting and grouping finding aids on certain themes and then index them by the topics of the portal (see chapter 5.2).
Creating a holdings or source guide is done in two steps: firstly you create the structure of the file; secondly you link the finding aids to this structure.
In the Content Manager, Holdings guide tab, you have a button to create Holdings guide that opens a dialog box in which you have to provide a minimum set of information for the new file: an identifier, a title and a description. Please note that the <eadid> of the file will be provided automatically by the Dashboard.
Figure 39a: Creation of a holdings guide: button in the content manager ...
Figure 39b: ... and dialog box to create the high level of description
After having completed and saved this form, you can "build" the structure of your holdings guide by creating levels and sublevels as you wish to. The same form (figure 39b) is used at each level and the structure displayed on the left part of the screen.
Figure 40a: Creating the structure of the holdings guide: each level and sublevel can be created, edited and suppressed via the option box on the right.
Figure 40b: For each level, the same dialog box and option box is always displayed
The different levels are to be seen as "node" where you can link your finding aids during the second step. Once saved, the HG is visible in the content manager. Its status is set to "Dynamic", and the linking is set to "0/0/0", indicating that there is no finding aid linked nor linkable yet.
Figure 41: Holdings guide in the content manager just after its creation: no finding aid linked
Please note that this creation is a one-shot: you cannot modify its structure or descriptions within the dashboard after having saved it. The only possibility for future changes (except linking finding aids) is to download the holdings guide and rework it in an XML editor.
In order to better display the archival content and use the advanced search tree, the finding aids should be linked to the holdings or source guide.
In the Content manager, select the files that you want to link, level by level, and choose "add finding aids to holdings guide" in the dropdown box list of actions. You can use the facilities of the content manager to order your files by title or by identifier. This order will be kept within the holdings guide.
The selected Finding aids are listed and you are asked to select certain options:
Figure 42: Linking the finding aids within the Dashboard
Be aware that - again - this is a one-shot: once linked you cannot unlink a finding aid. But you can add new finding aids once uploaded in the Dashboard, if your structure has all needed levels and sub-levels. You can check the result in the Content manager: the linked finding aids are noted in the Holdings guide column with the corresponding number (1 or more if one finding aid is linked to a source guide also for instance).
Figure 43: Indication of the links within the content manager: now set to 2/2/2
You can also check the result in the portal: the advanced search tree shows the hierarchy between the holdings guide and the finding aids. The preview contains two links and allows the user to either display directly the finding aids, either display the holdings guide open on this finding aids (see figure 10).
For more information on the steps needed to deliver your content to Europeana, please see: http://www.apex-project.eu/index.php/en/component/docman/doc_download/302-europeana-steps-to-deliver-data?Itemid=185.
Detailed explanation on the conversion and mapping done from apeEAD fromat to EDM (Europeana Data Model) can be found also on the APEx project website.
Figure 44: Convert the data to the Europeana data model (EDM format): set your preferences for the automatic conversion script
See also chapter 4.2; the options are the same as in the profile section.
Once the options chosen, a click on the convert button send you back to the content manager while the file is put in the queue in order to be processed.
To actually deliver content to Europeana, the Action "deliver EDM to Europeana" has to be chosen in the content manager. The converted data will then be placed in the OAI-PMH repository of the Archives Portal Europe.
Figure 45: Status of the EDM files in the content manager: the column "EDM" gives the number of converted files (number of CHO - cultural heritage object - and linked images);
the column "Europeana" indicates if the files have been placed in the OAI repository
A "data quality report" on the EDM can be displayed if wished; it focuses on the identifiers.
Figure 46: Checks on the identifiers
Please note that Europeana harvests the Archives Portal Europe only once a month (on the 21st) and then further processes the data before actually publishing it on their portal. Therefore, it can take up to two months before seeing your data online. There is unfortunately no possibility to have a preview of the result of the conversion so far.